Skip to content

Fix doris 26821 map lazy fill#65403

Open
HappenLee wants to merge 2 commits into
apache:masterfrom
HappenLee:fix-doris-26821-map-lazy-fill
Open

Fix doris 26821 map lazy fill#65403
HappenLee wants to merge 2 commits into
apache:masterfrom
HappenLee:fix-doris-26821-map-lazy-fill

Conversation

@HappenLee

@HappenLee HappenLee commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

HappenLee added 2 commits July 7, 2026 22:17
@HappenLee
[fix](be) Fill skipped map side during lazy pruned reads
75fe1db 
### What problem does this PR solve?

Issue Number: None

Related PR: None

Problem Summary: Lazy pruned complex-column reads can materialize only KEYS or VALUES for a MAP nested under a STRUCT when predicate evaluation reads another path. The skipped MAP side stayed empty in the lazy phase, so ColumnMap offsets no longer matched key/value column sizes. This patch fills defaults for a skipped MAP child according to the element count while leaving predicate-materialized children untouched.

### Release note

Fix MAP nested column pruning with lazy materialization when only map keys or values are projected.

### Check List (For Author)

- Test: Unit Test: ./run-be-ut.sh -j 48 --run --filter=ColumnReaderTest.MapNextBatchLazyFillsSkippedKeyOrValueColumn:ColumnReaderTest.MapReadByRowidsLazyFillsSkippedKeyOrValueColumn
- Test: Regression test: ./run-regression-test.sh --run --conf /tmp/doris_26821_regression-conf.groovy -d datatype_p0/complex_types -s test_pruned_struct_map_lazy
- Test: Static analysis: CLANG_TIDY_BINARY=/mnt/disk6/common/ldb_toolchain_028/bin/clang-tidy build-support/run-clang-tidy.sh --files be/src/storage/segment/column_reader.cpp be/test/storage/segment/column_reader_test.cpp --build-dir be/ut_build_ASAN
- Behavior changed: Yes. MAP nested lazy materialization now keeps key/value child sizes aligned when one side is pruned.
- Does this need documentation: No
@HappenLee
[improvement](be) Add native ZoneMap serialization
bec1331 
### What problem does this PR solve?

Issue Number: None

Related PR: None

Problem Summary: ZoneMap min and max values were serialized through type-specific OLAP strings, and page ZoneMaps were stored as one serialized ZoneMapPB per page in an IndexedColumn. This added string formatting/parsing cost and per-page protobuf/container overhead. This change adds native binary ZoneMapEntryPB serialization for segment ZoneMaps, stores page ZoneMaps in one array-oriented native ZoneMapPagesPB payload, keeps legacy readers for old rowsets, and switches page pruning paths to use decoded ZoneMap objects directly.

### Release note

New rowsets store ZoneMap metadata in a native binary format. Existing rowsets with legacy string ZoneMap metadata remain readable.

### Check List (For Author)

- Test: Regression test / Unit Test / Manual test
    - Unit Test: ./run-be-ut.sh -j 48 --run --filter=ColumnZoneMapTest.*
    - Unit Test: ./run-be-ut.sh -j 48 --run --filter=ColumnZoneMapTest.*:SegmentIteratorExprZonemapTest.*
    - Unit Test: ./run-be-ut.sh -j 48 --run --filter=MetadataAdderTest.* (meta_load_with_pb_test passed; metadata_adder_test failed on pre-existing global MetadataAdder counters)
    - Manual test: PATH=/mnt/disk6/common/ldb_toolchain_toucan/bin:$PATH build-support/check-format.sh
    - Manual test: build-support/run-clang-tidy.sh --build-dir be/ut_build_ASAN --files be/src/storage/index/zone_map/zone_map_index.cpp be/src/storage/segment/column_reader.cpp be/src/storage/segment/segment_iterator.cpp be/test/storage/segment/zone_map_index_test.cpp be/test/storage/metadata_adder_test.cpp
    - Manual test: git diff --check
- Behavior changed: Yes. New rowsets no longer dual-write legacy ZoneMap metadata; pruning semantics are unchanged.
- Does this need documentation: No
@hello-stephen

hello-stephen commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@HappenLee

HappenLee commented Jul 9, 2026

Copy link
Copy Markdown
Contributor Author

run buildall

@hello-stephen

hello-stephen commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

FE UT Coverage Report

Increment line coverage `` 🎉
Increment coverage report
Complete coverage report

@hello-stephen

hello-stephen commented Jul 9, 2026

Copy link
Copy Markdown
Contributor
TPC-H: Total hot run time: 29528 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit bec1331b78dfc5a1c0ff623b1ab490d2e8200afa, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17647	4051	4006	4006
q2	2031	331	214	214
q3	10260	1484	848	848
q4	4685	486	334	334
q5	7529	920	574	574
q6	183	166	134	134
q7	758	837	618	618
q8	9413	1585	1660	1585
q9	5685	4436	4455	4436
q10	6705	1821	1568	1568
q11	505	346	316	316
q12	728	546	438	438
q13	18117	3870	2759	2759
q14	260	262	241	241
q15	q16	791	788	707	707
q17	1021	1041	872	872
q18	6799	5706	5622	5622
q19	1167	1148	1062	1062
q20	706	650	513	513
q21	5237	2623	2381	2381
q22	439	354	300	300
Total cold run time: 100666 ms
Total hot run time: 29528 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	4391	4237	4298	4237
q2	287	324	223	223
q3	4574	5032	4433	4433
q4	2039	2136	1339	1339
q5	4429	4305	4321	4305
q6	236	175	128	128
q7	1751	1611	2161	1611
q8	2580	2216	2206	2206
q9	8061	8193	7897	7897
q10	4772	4741	4326	4326
q11	617	438	402	402
q12	774	756	556	556
q13	3259	3681	2959	2959
q14	291	299	282	282
q15	q16	714	749	662	662
q17	1347	1313	1308	1308
q18	8311	7410	7125	7125
q19	1165	1113	1156	1113
q20	2203	2232	1948	1948
q21	5174	4571	4408	4408
q22	507	460	405	405
Total cold run time: 57482 ms
Total hot run time: 51873 ms

@hello-stephen

hello-stephen commented Jul 9, 2026

Copy link
Copy Markdown
Contributor
TPC-DS: Total hot run time: 180640 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit bec1331b78dfc5a1c0ff623b1ab490d2e8200afa, data reload: false

query5	4317	653	506	506
query6	510	232	209	209
query7	4832	588	354	354
query8	339	192	176	176
query9	8793	4089	4101	4089
query10	485	359	322	322
query11	5953	2359	2144	2144
query12	154	105	104	104
query13	1310	594	466	466
query14	6284	5315	5042	5042
query14_1	4293	4297	4308	4297
query15	222	206	189	189
query16	1054	481	464	464
query17	1137	723	589	589
query18	2672	485	353	353
query19	211	205	151	151
query20	117	115	110	110
query21	244	161	135	135
query22	13631	13511	13516	13511
query23	17393	16459	16208	16208
query23_1	16431	16225	16275	16225
query24	7677	1780	1289	1289
query24_1	1330	1311	1313	1311
query25	584	473	406	406
query26	1326	354	218	218
query27	2596	586	390	390
query28	4445	2045	2039	2039
query29	1085	633	513	513
query30	334	267	230	230
query31	1138	1102	983	983
query32	106	64	62	62
query33	547	323	261	261
query34	1194	1136	658	658
query35	763	787	689	689
query36	1415	1356	1269	1269
query37	153	103	93	93
query38	1864	1698	1648	1648
query39	927	916	880	880
query39_1	885	878	889	878
query40	241	156	137	137
query41	66	62	62	62
query42	93	90	94	90
query43	318	334	278	278
query44	1416	776	786	776
query45	200	186	182	182
query46	1076	1217	751	751
query47	2351	2345	2241	2241
query48	387	421	304	304
query49	586	424	314	314
query50	1097	420	333	333
query51	10646	10540	10767	10540
query52	93	86	75	75
query53	266	274	206	206
query54	289	237	217	217
query55	78	75	66	66
query56	286	291	293	291
query57	1436	1428	1308	1308
query58	283	263	256	256
query59	1603	1618	1484	1484
query60	302	278	250	250
query61	149	145	153	145
query62	712	642	592	592
query63	252	205	211	205
query64	2785	1063	874	874
query65	4844	4816	4796	4796
query66	1787	522	385	385
query67	29599	29503	29278	29278
query68	3327	1587	987	987
query69	441	308	273	273
query70	1083	958	979	958
query71	355	340	308	308
query72	3069	2674	2410	2410
query73	860	780	433	433
query74	5186	4953	4797	4797
query75	2618	2577	2218	2218
query76	2347	1187	774	774
query77	367	387	279	279
query78	12323	12503	11708	11708
query79	1437	1143	746	746
query80	1289	530	473	473
query81	560	334	281	281
query82	626	159	125	125
query83	386	313	296	296
query84	278	161	134	134
query85	954	613	516	516
query86	436	286	273	273
query87	1834	1826	1727	1727
query88	3710	2832	2788	2788
query89	452	406	359	359
query90	1927	202	198	198
query91	204	187	162	162
query92	65	65	56	56
query93	1683	1497	928	928
query94	725	372	324	324
query95	766	576	484	484
query96	1042	798	358	358
query97	2730	2692	2565	2565
query98	217	209	203	203
query99	1149	1183	1022	1022
Total cold run time: 266897 ms
Total hot run time: 180640 ms

@hello-stephen

hello-stephen commented Jul 9, 2026

Copy link
Copy Markdown
Contributor
ClickBench: Total hot run time: 25.28 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit bec1331b78dfc5a1c0ff623b1ab490d2e8200afa, data reload: false

query1	0.00	0.00	0.01
query2	0.09	0.05	0.05
query3	0.29	0.13	0.14
query4	1.60	0.14	0.14
query5	0.24	0.25	0.24
query6	1.29	1.07	1.10
query7	0.04	0.01	0.01
query8	0.06	0.04	0.04
query9	0.39	0.33	0.32
query10	0.57	0.56	0.55
query11	0.20	0.14	0.14
query12	0.19	0.14	0.14
query13	0.46	0.47	0.49
query14	1.03	1.02	1.01
query15	0.61	0.58	0.60
query16	0.31	0.33	0.32
query17	1.11	1.15	1.09
query18	0.23	0.21	0.21
query19	2.02	1.99	1.96
query20	0.02	0.01	0.02
query21	15.44	0.20	0.12
query22	4.84	0.04	0.04
query23	16.17	0.29	0.12
query24	3.02	0.41	0.33
query25	0.10	0.06	0.04
query26	0.71	0.20	0.15
query27	0.03	0.05	0.04
query28	3.51	0.92	0.54
query29	12.50	4.15	3.27
query30	0.27	0.15	0.15
query31	2.77	0.59	0.32
query32	3.22	0.60	0.49
query33	3.14	3.30	3.33
query34	15.61	4.20	3.57
query35	3.57	3.55	3.51
query36	0.56	0.45	0.43
query37	0.09	0.07	0.07
query38	0.05	0.04	0.04
query39	0.04	0.03	0.03
query40	0.18	0.17	0.15
query41	0.08	0.03	0.03
query42	0.04	0.03	0.03
query43	0.05	0.04	0.03
Total cold run time: 96.74 s
Total hot run time: 25.28 s

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@yiguolei yiguolei Awaiting requested review from yiguolei yiguolei is a code owner
@liaoxin01 liaoxin01 Awaiting requested review from liaoxin01 liaoxin01 is a code owner
@gavinchou gavinchou Awaiting requested review from gavinchou gavinchou is a code owner
@eldenmoon eldenmoon Awaiting requested review from eldenmoon eldenmoon is a code owner
@csun5285 csun5285 Awaiting requested review from csun5285 csun5285 is a code owner
@airborne12 airborne12 Awaiting requested review from airborne12 airborne12 is a code owner

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@HappenLee @hello-stephen